Comparing Comparisons: Document Clustering Evaluation Using Two Manual Classifications

“Describe your occupation in a few words”, is a question answered by 44 000 Swedish twins. Each respondent was then manually categorized according to two established occupation classification systems. Would a clustering algorithm have produced satisfactory results? Usually, this question cannot be answered. The existing quality measures will tell us how much the algorithmic clustering deviates from the manual classification, not if this is an acceptable deviation. But in our situation, with two different manual classifications (in classification systems called AMSYK and YK80), we can indeed construct such quality measures. If the algorithmic result differs no more from the manual classifications than these differ from each other (comparing the comparisons) we have an indication of its being useful. Further, we use the kappa coefficient as a clustering quality measure. Using one manual classification as a coding scheme we assess the agreement of a clustering and the other. After applying both these novel evaluation methods we conclude that our clusterings are useful.